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ABSTRACT 

The development of a needs instrument designed to 
assess the continuing education needs of school administrators is 
described. The instrument was constructed to assess the 
administrators' continuing needs in: (1) principal ship ; (2) the 
school as a system; (3) problem solving; (4) staff development; (5) 
collaborative long-range planning; (6) short-term planning; and (7) 
personal awareness. Thirty-eight questions for the assessment were 
written using a standard domain sampling model, and the revised 
questions were made into a questionnaire using a Likert discrepancy 
model. Respondents rated statements on a five-point scale of 
perception of the degree to which the situation actually exists and 
perception of the extent to which it is desired to exist. The 
discrepancy between the two scale values is referred to as the need 
index. Content validity was established through factor analysis with 
a sample of 191 school administrators. Reliability measures indicate 
that the instrument can be used for group assessment. Items for the 
survey are listed, and two tables present the varimax rotated factor 
matrix for the two instrument columns (Contains 34 references.) 
(SLD) 



y? it iV Vf it it it it it it it it it it it it it it it it it it it it ie it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it is it it it it it it is it it it it it 

Reproductions supplied by EDRS are the best that can be made '"^ 
''^ from the original document. '"^ 

it iticit it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it i< it it it it it it it it it it it it iK it it it it it it it it it it 



us DEPARTMENT OF EDUCATION 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



E0U(?ATlONAt. RESOURCES INFORMATION- 



CENTER (ERiC 



Tnis documeni has Dee" 'eptoaucet^ as 
Oi.fl.nat.ng .1 



C Mioo* changes have D«»eh i>a(3e inipiove 
tep'Oducl'Oh auai'ty 



• Poinisoi v.e* o' opth.ons Slated «h In. s doc u 
•n.?n) do hoi hecessaf'iy 'epfesP"i o"i'-iai 
OEP' po&it.on o' policy 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



DEVELOPING AN ADMINISTRATIVE ASSESSMENT INSTRUMENT 



WILLIAM L. JOHNSON, Ed.D. 
DEAN OF FACULTY 
CHAIR, EDUCATION & PSYCHOLOGY 
AMBASSADOR COLLEGE 
BIG SANDY, TEXAS 75755 



KAROLYN J. SNYDER, Ed.D. 
DIRECTOR, SCHOOL MANAGEMENT INSTITUTE 
UNIVERSITY OF SOUTH FLORIDA 
TAMPA, FLORIDA 33620 



ANNABEL M. JOHNSON, Ph.D. 
ADJUNCT PROFESSOR OF HOME ECONOMICS 
AMBASSADOR COLLEGE 
BIG SANDY, TEXAS 75755 



mmmmmi 



ABSTRACT 

Educators are generating powerful ideas and practices to help education be more 
successful. In this document, the authors discuss the development of a needs instrument 
designed to assess the continuing education needs of school administrators. 
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During the past decade, Snyder and Anderson implemented a leadership training 
program know as Managing Productive Schools (MPS) in Florida, Minnesota, and Virginia. 
They have trained several thousand administrators worldwide. The program is based on an 
extensive research base and also on a systems approach to organizational development. The 
books by Snyder and Anderson (1986) and Snyder (1988) explicitly outline their model for 
leadership training. 

The model assumes that schools that are seeking to transform themselves and to solve 
major challenges have a vision of a great school that is shared by all members. In addition, 
an annual school enhancement plan is designed each year and becomes a focus for work 
activity. A rich array of professional development opportunities that center around 
enhancement goals is provided for all levels of staff. The central thrust of school 
enhancement is to align the instructional program routinely to address the needs of various 
student populations. Allocating resources where needs are the greatest creates natural new 
structures of work and learning. Resources, information, and opportunity are fundamental 
materials for organizational productivity (Johnson, Snyder, & Johnson, 1991). Progress with 
goals, and toward better meeting client needs, is assessed routinely to provide feedback and 
feed-forward information. 

Considering the research base, an instrument was constructed to assess administrators' 
continuing education desires in seven specified areas: (a) the principalship, (b) the school 
as a system, (c) problem solving, (d) staff development, (e) collaborative long-range 
planning, (f) short-term planning, and (g) personal awareness. These seven areas appeared 
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in the literature review to be important dimensions pertaining to culture, planning, 
professional development, program development, and assessment. 

These subscales, a brief content description, and a sample of an item statement 
follow: principalship consisting of questions embracing the perspectives of leadership, 
planning, organizing, and motivating organizational behavior. Next was the topic of the 
school as a system. Questions addressed the environmental suprasystem as well as the goal, 
psychosocial, structural, technical, and management subsystems. To our knowledge, there 
have been no empirical studies that have accessed the specific effects of school subsystems 
on school performance. 

The third subscale was problem solving focusing on real school organizational 
problems from the perspective of roles, communication skills, and problem solving steps. 
Next was short-term planning, which focused on the principal's varied tasks such as action 
planning and force field analysis. The fourth subscale addressed was staff development 
encompassing clinical supervision, in-service training, and evaluation. The fifth subscale was 
collaborative long-range planning for school improvement which included goal setting, 
resource assessment, program planning, and implementation, and monitoring functions and 
evaluation procedures. Finally, there was a personal awareness subscale embracing the 
perspectives of personality characteristics, self-concept, and administrative-style effectiveness. 
The 38 questions that were written were then randomly assigned to the instrument. 

The construction and administration of the instrument involved the following steps: 
(a) development of a pool of subscaled questions; (b) development of questionnaire forms 
and a pool of instrument questions; (c) selection of administrative personnel to respond to 
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the scale; (d) actual administration of the scale; (e) analysis of the scale for the selection of 
the pool of items to be included in the instrument; (f) development of computer programs 
for instrument scoring and classification procedures; and (g) reliability and validity 
assessments of the scale for the pilot and final versions. 

Development of Items 

The 38 questions were written using the standard domain sampling model supported 
by Nunnally (1967) and Cronbach (1970). Initial questions addressing the seven major 
training areas were written. These questions were scrutinized and studied. Next the pool 
of items was expanded. Finally the questions were rewritten and revised. These final 
revisions produced the pool of items contained in the Survey for Administrative Workshop 
instrument. 

Next a questionnaire form was developed for the pool of items. A Likert format was 
employed as the response mode for all questions. Lemon (1973) has stated that much of 
the experimental research pertaining to attitude change involves measuring attitudes by some 
simple, easily administered instrument, usually a rating-scale format, and then attributing 
changes or differences in the ratings to the experimental manipulations. 

In this context, the Likert scale of summative ratings is the most widely used attitude 
measure in current research. Gronlund (1976) stated that Likert's approach to attitude 
scaling was less time consuming in development than earlier methods such as Thurston's and 
Remmers'. Borg and Gall (1974) identified the Likert technique as the easiest method 
presently available for developing scales needed in attitude research. 



McGinnies (1970) clarified the procedures involved in constructing a Likert scale. 
The first step involves the writing of a number of statements dealing with the topic under 
investigation. Respondents then indicate their reaction to each question on an interval scale, 

Severy (1974) added that it is assumed all written questions reflect the same 
attitudinal dimensions when a Likert scale is developed. Advantages of the Likert scale are: 
(1) that construction is less time consuming and easier than other scaling techniques, and (2) 
that fewer items are required to obtain higher correlation coefficients (Ahmann <& Glock, 
1975; Borg & Gall, 1974; Gronlund, 1976; Lemon, 1973; Shaw & Wright, 1967). 

Shaw and Wright (1967) indicated that items should be written in simple and clear 
language, should contain a single idea, and should be unambiguous, Borg and Gall (1974) 
noted that an instrument dealing with attitudes would generally be constructed as an attitude 
scale and should include at least ten items to obtain a valid picture of the attitude 
concerned. Tittle and Hill (1967) investigated the predictive efficiency of four frequently 
used attitude measurement techniques in terms of predicting indices of voter behavior. The 
Thurston, Guttman, Likert, and semantic differential scales were compared. In Tittle and 
Hill's study, the Likert scale was superior to all other scales in predictive validity. Their 
evidence suggested that in cases where the Likert and Thurston scales were of equal length, 
the Likert scale exhibited higher reliability, 

Likert scales are self-reporting tests commonly used in studies which include measures 
of attitudes. Gronlund (1976) has observed that one of the most severe drawbacks of such 
is the distortion of answers by respondents. Attitude scales, therefore, are most useful when 
the participants have little reason for falsification. 
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Borg and Gall (1974) have written that individuals who are planning to collect 
information about attitudes should first search the literature to determine if a suitable scale 
has already been constructed. If such a scale was not available, it would be necessary to 
develop one. 

A Likert discrepancy model was chosen because of its applicability in general or first- 
time assessment trials (Witkin, 1977). Discrepancy as defined by Tyler (1949) is understood 
to mean a difference between some standard of value and an actual status. Sarthory (1977) 
has defined a need as a quantifiable gap in attitude, achievement, or performance between 
the ideal and the real. 

Kaufman (1972) and Southard (1974) have identified the four specific components 
in a discrepancy needs assessment procedure: (1) determine desired conditions, (2) 
determine existing conditions, (3) analyze discrepancies between the actual and desired 
conditions, and (4) assign priorities to the discrepancies. These discrepancies constitute 
indices of need. 

Pertaining to the discrepancy format, respondents rate specific statements on two five- 
point Likert scales: (a) perception of the degree to which the situation actually exists (A) 
and (b) perception of the extent to which the situation is desired to exist (D). The 
discrepancy between the two scale values, for a specific question, is referrer" to as the need 
index. 

The discrepancy format response mode included the following two column response 
categories: (a) actual competence (ACT) (consisting of the following scale): (1) no 
knowledge of; (2) knowledge of; (3) work with; (4) extensive work with; (5) expertise in; and 
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(b) desired training (DES) (consisting of the following scale): (1) desire no training, skill 
unrelated; (2) desire no training, competency high; (3) desire awareness session; (4) desire 
training; and (5) desire further training. The competency column is the actual status while 
the training column is the desired status Jenkins and Taber (1977) found in an 
agree/disagree context that the number of . jsponse categories above five did not, in any 
\ situation, yield a significant increase in Likert discriminability. 

In recent years a great deal of research has been done to establish the fact that 
response sets do exist in questionnaire responses (Bendig, 1962; Hand, 1964; Hand & 
Brazzell, 1965; Rosenwald, 1961; Ruebush, 1963). In order to avoid response set bias or 
acquiescence tendency, the try-out questions were randomly assigned within the instrument 
(Cronbach, 1970). Many researchers believe that response sets are only a mild threat to 
valid measurement, and that their importance has been overestimated. The available 
evidence he asserts does not justify the strong negative assertions made by response-set 
enthusiasts. 

There is the question of a metric base for the variables. Stevens (1946) wrote that 
factor analysis requires the variables be measured on at least an interval level. This 
requirement is implied by the use of correlation or covariance matrices as the basic input 
to factor analysis. If it were argued that the scaling used was not absolutely of a metric base, 
would a researcher always avoid factor analysis? Kim and Mueller (1978) wrote that such 
is not necessarily so. Many variables, such as measure of attitudes and opinions, do not have 
a clearly established metric base. Furthermore, it is generally assumed that many ordinal 
variables may be given numeric values without distorting the underlying properties. The 
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final answer to the question of applying multivariate analysis to a nonmetric variable base 
really hinges on two considerations: (a) how well the arbitrarily assigned numbers reflect 
the underlying true distances, and (b) the amount of distortion introduced in the correlation 
procedures (which become the basic input to factor analysis) by the distortions of scaling. 
Another consideration not mentioned by Kim and Mueller (1978) would be the importance 
of precision of measurement. From a post hoc research purist perspective, it would have 
been better to have omitted the desire no (raining (competency high) response choice. 
However, very few of the respondents chose this category. Fortunately, correlation 
coefficients are fairly robust with respect to ordinal distortions in measurement (Kim, 1975; 
Labovitz, 1967, 1970). Hence, as long as the distortions introduced by assigning numeric 
values to ordinal categories are not very substantial, treating ordinal variables as if they are 
metric variables can be justified (Kim & Mueller, 1978). The scale descriptions were written 
to reflect as near a metric base as possible. 

Development of the Instrument Form 

An introductory paragraph in the questionnaire explained the purpose of the 
instrument distribution. Immediately following were directions and definitions pertaining to 
the completion of the scale and meanings of the terms used. A concluding demographic 
section asked for workshop training preferences and data pertaining to each participant's 
title, division of responsibility, school-district size and setting, major economic base, and 
community ethnic composition. 

Selection of Respondents and Test Scoring 
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All the respondents were administrative-type personnel, such as principals, assistant 
principals, superintendents, assistant superintendents, supervisors, or curriculum directors. 
School districts included both the Northeast and Northside Independent School Districts in 
San Antonio, Texas; the Penn-Harris-Madison School Corporation in Osceola, Indiana; and 
the Princeton Regional Schools in Princeton, New Jersey. The total sample size was 191. 

A discrepancy format instrument like this would be scored using a correlated t test 
to calculate a needs index for each question. The literature of statistical procedures is 
replete with descriptions of and derivations for this procedure (Hicks, 1973; Hinkle & 
Wiersma, 1979; Kirk, 1968). 

Analysis of the Instrument 

Construct validity was established through the factor analysis of the final 38-item scale 
(Gorsuch, 1983). The factoring method used was PA2 (principal factoring using iteration) 
with the varimax criterion specified as the method of orthogonal rotation (Nie, Hull, Jenkins, 
Steinbrenner, & Bent, 1975). Three present status scales (PRE) and three desired status 
scales (DBS) emerged. The PRE and DES factors were paired and common questions were 
extracted. This procedure reduces the common factors to two. The PRE questions for 
factor one (factor loadings greater than or equal to .4) were PRE 16, 17, 18, 19, 21, 23, 24, 
28, 31, 32, 35, and 36. Cronbach's alpha was .94 for this subscale. The corresponding DES 
questions had an alpha value of .94. Both PRE and DES questions pooled had an alpha 
value of .88. The identical procedure was followed in investigating the psychometric 
properties of the second factor; the PRE questions which emerged were PRE 1, 2, 3, 5, 8, 
12, and 29. Cronbach's alpha was .87. The corresponding DES questions had an alpha 
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value of .91. The pooled set of PRE and DES questions had an alpha value of .80. All 
PRE questions (1, 2, 3, 5, 8, 12, 16, 17, 18, 19, 21, 23, 24, 28, 29, 31, 32, 35, and 36) had an 
alpha value of .95. The alpha value for all the corresponding DES questions was .96. All 
PRE and DES questions pooled had an alpha value of .91. Tables 1 and 2 list the varimax 
rotated factor matrices for the PRE and DES columnr. (only items with factor loadings 
greater than or equal to A were considered for inclusion in the instrument). 

Questions 1, 2, 3, 5, 8, 12, and 29 dealt with planning and staff development. 
Questions 16, 17, 18, 19, 21, 23, 24, 28, 31, 32, 35, and 36 dealt with assessment and 
motivation. The administrators wanted to know how to assess the effectiveness of their 
staffs, how to set goals, and how to motivate their staff to work at their peak and accomplish 
their goals. 

Summary and Conclusion 

This article illustrates the process for the development and field testing of a needs 
assessment instrument for the assessment of the continuing educational needs of 
administrators. The process linked the literature review with a primary factor analysis. The 
instrument was shown to be reliable and valid. Because of the reliability measures, the 
instrument can be used for group assessment. If it were desired that norms be established 
in an instrument development study, such would involve the use of converting and equating 
procedures to generate a normal distribution process would allow the generation of 
percentile reference points for the interpretation of a^ ^ninistrators' scores. A sample could 
be checked with respect to geographical distribution, socioeconomic level or other 
information relevant to the study. For this particular study, however, norms were not 
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developed. Such could be a point-of-departure for another study which would encompass 
a separate body of psychometric literature and statistical analyses. 



INSTRUMENT ITEMS FOR SURVEY 

1. How to design inservice programs which increase the effectiveness of my staff. 

2. How to lead my staff and parents in long-range planning. 

3. How to develop collaborative action plans for accomplishing school goals. 

4. How to lead groups in creative problem solving activities. 

5. How to develop effective action plans for accomplishing school goals. 

6. How to think about my school (school system) from a systems approach to 
org^mization and management. 

7. How to provide leadership to my school in its continuous improvement. 

8. How to plan for change and to be responsible to the emerging needs of my staff, 
students, and parents. 

9. How to conduct formative evaluation. 

10. How to conduct summalive evaluation. 

11. How to analyze my school organisation and its potential for growth. 

12. How to plan for anc^ conduct staff evaluation. 

13. How to think about my school as an ecosystem and the various operational subsystem 
in my school environment. 

14. How to set goals collaboratively. 

15. How to involve teachers in peer supervision and continuous improvement on the job. 

16. How to assess my administration effectiveness in working with my staff, students, and 
parents to improve my school. 

17. How to involve teachers, students, and parents in planning for school improvement. 

18. How to assess the needs of my school and set my own goals for improvement. 

19. How to motivate my staff to work at the peak of their potential. 

20. How to control all school activities so that we accomplish our goals effectively. 

O Id 
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21. How to assess the growth needs of my facuhy. 

22. How to use creative problem solving techniques in dealing with specific school 
organizational problems. 

23. How to assess my own beliefs and values in relation to what exists in my school 
currently and to my visions fcr improvement. 

24. How to assess the various forces within and outside my school and their influence on 
my goals for improvement. 

25. How to conduct long-range planning for school improvement. 

26. How to generate creative options for accomplishing goals. 

27. How to evaluate the accomplishment of school goals. 

28. How to identify my own personal characteristics and their effect on my performance. 

29. Hew to monitor the implementation of school improvement plans. 

30. How to plan for my own professional growth needs. 

31. How to assess our resources for all goals. 

32. How to develop creative solutions to school problems. 

33. How to assess my self-concept and its effect on my goals. 

34. How to nurture creativity among staff members. 

35. How to organize my school so as to maximize our resources in accomplishing our 
purposes. 

36. How to increase my visions of the possibilities for educational change in my school. 

37. How to increase my own skills in supervision through observation of teaching 
performance, critical analysis, and interpretation of data and data feedback and 
recommendations for improvement. 

38. How to prioritize goal options. 
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TABLE 1 



VARIMAX ROTATED FACTOR MATRIX BY QUESTION FOR 
THE PRE COLUMN IN THE INSTRUMENT 



Item Number* 


Factor #1 


Factor #2 


Factor #3 




(78.9)" 


(6.7) 


(4.3) 


PRE 01 


0.221 


0.577 


0.121 


PRE 02 


0.288 


0.634 


0.190 


PRE 03 


0.221 


0.519 


0.099 


PRE 04 


0.091 


0.529 


0.180 


PRE 05 


0.316 


0.616 


0.153 


PRE 06 


0.199 


0.328 


0.208 


PRE 07 


0.397 


0.678 


0.168 


PRE 08 


0.355 


0.594 


0.260 


PRE 09 


0.169 


0.320 


0.230 


PRE 10 


0.2S8 


0.298 


0.158 


PRE 11 


0.616 


0.334 


-0.040 


PRE 12 


0.296 


0.535 


0.132 


PRE 13 


0.280 


0.176 


0.115 


PRE 14 


0.400 


0.477 


0.298 


PRE 15 


0.379 


0.106 


0.212 


PRE 16 


0.632 


0.144 


0.253 


PRE 17 


0.571 


0.335 


0.260 


PRE 18 


0.603 


0.354 


0.292 


PRE 19 


0.592 


0.345 


0.287 


PRE 20 


0.682 


0.297 


0.256 


PRE 21 


0.649 


0.167 


0.229 


PRE 22 


0.620 


0.266 


0.180 


PRE 23 


0.530 


0.166 


0.365 


PRE 24 


0.630 


0.143 


0.290 


DO TZ? 1C 
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PRE 26 


0.547 


0.312 


0.243 


PRE 27 


0.683 


0.377 


0.198 


PRE 28 


0.549 


0.104 


0.541 


PRE 29 


0.560 


0.407 


0.325 


PRE 30 


0.249 


0.267 


0.680 


PRE 31 


0.615 


0.194 


0.360 


PRE 32 


0.581 


0.294 


0.296 


PRE 33 


0.352 


0.218 


0.724 


PRE 34 


0.392 


0.226 


0.502 


PRE 35 


0.624 


0.282 


0.349 
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TABLE 1 - Continued 



VARIMAX ROTATED FACTOR MATRIX BY QUESTION FOR 
THE PRE COLUMN IN THE INSTRUMENT 



Item Number^ 


Factor #1 


Factor #2 


Factor #3 




(78.9)" 


(6.7) 


(4.3) 


PRE 36 


0.662 


0.289 


0308 


PRE 37 


0.428 


0.348 


0.471 


PRE 38 


0.552 


0.242 


0.455 



^ Items in present-competency column (PRE) correspond to the instrument 
numbering sequence 



^ Percent of total variance is in parentheses 



TABLE 2 



VARIMAX ROTATED FACTOR MATRIX BY QUESTION FOR 
THE DES COLUMN IN THE INSTRUMENT 



Item Number^ 




Factor #1 
(83.1)'' 


Factor #2 
(5.4) 


Factor #3 

/A O \ 

(4.3) 


DES 01 




0.231 


0.512 


0.105 


DES 02 




0.182 


0.578 


0.175 


DES 03 




0.120 


0.458 


0.479 


DES 04 




0.203 


0.176 


0.258 


DES 05 




0.202 


0.412 


0.258 


DES 06 




0.198 


0.246 


0.626 


DES 07 




0.332 


0.249 


0.284 


DES 08 




0.322 


0.464 


0.220 


DES 09 




0.163 


0.671 


0.330 


DES 10 




0.185 


0.594 


0.357 


DES 11 




0.386 


0.454 


0.512 


DES 12 




0.272 


0.609 


0.248 


DES 13 




0.101 


0.154 


0.687 


DES 14 




0.270 


0.339 


0.503 


DES 15 




0.244 


0.435 


0.165 


DES 16 




0.598 


0.427 


0.043 


DES 17 




0.408 


0.514 


0.092 


DES 18 




0.504 


0.438 


0.361 


DES 19 




0.540 


0.391 


0.076 


DES 20 




0.348 


0.292 


0.335 


DES 21 




0.640 


0.289 


0.151 


DES 22 




0.289 


0.272 


0.225 


DES 23 




0.510 


0.304 


C\ A ^ A 

0.464 


DES 24 




0.474 


0.173 


0.541 


DES 25 




0.335 


0.477 


0.27U 


DES 26 




0.223 


0.198 


0.373 


DES 27 




0.271 


0.506 


0.199 


DES 28 




0.571 


0.318 


0.317 


DES 29 




0.266 


0.428 


0.223 


DES 30 




0.521 


0.287 


0.266 


DES 31 




0.413 


0.296 


0.349 


DES 32 




0.408 


0.165 


0.236 


DES 33 




0.565 


0.159 


0.436 


DES 34 




0.548 


0.157 


0.179 


DES 35 




0.537 


0.200 


0.388 



(Continued) 
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TABLE 2 — Continued 

VARIMAX ROTATED FACTOR MATRIX BY QUESTION FOR 
THE DES COLUMN IN THE INSTRUMENT 



Item Number" Factor #1 Factor #2 Factor #3 

(83.1)" (5.4) (4.3) 



DES 36 0.619 0.084 0.479 

DES 37 0.366 0.462 0.063 

DES 38 0.333 0.420 0.444 



^ Items in desired-competency column (DES) correspond to the instrument 
numbering sequence 

Percent of total variance is in parentheses 
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